Data Engineering Path · Airflow
Airflow Core Glossary
📖
Quick Reference — Terms You'll Use Every Day
This glossary covers the essential Airflow terminology. Bookmark this page — these terms appear in every Airflow conversation, interview, and debugging session.
DAG Components
| Term | Definition | Code Reference |
|---|---|---|
| DAG | A collection of tasks with defined dependencies. The "blueprint" of your workflow. | DAG(dag_id="my_pipeline") |
| Task | A single unit of work in a DAG. Implemented as an Operator instance. | PythonOperator(task_id="extract") |
| Task Instance | A specific occurrence of a task — a task + a DAG Run + a logical date. | Visible in the Airflow UI Grid view |
| Operator | A class that defines a single task. Operators determine what gets done. | BashOperator, PythonOperator |
| Sensor | A special operator that waits for an external condition to be met before proceeding. | S3KeySensor, HttpSensor |
Execution Concepts
| Term | Definition | Example |
|---|---|---|
| DAG Run | An instantiation of a DAG at a specific logical date. | sales_etl @ 2024-01-15T06:00:00 |
| Logical Date | The date/time that a DAG Run is "logically" associated with (previously execution_date). |
{{ ds }} in Jinja templates |
| Schedule | The interval at which DAG Runs are created. | "0 6 * * *" (daily at 6 AM) |
| Backfill | Running a DAG for past logical dates that were missed. | airflow dags backfill -s 2024-01-01 |
| Catchup | Automatically backfill DAG Runs from start_date to now. |
catchup=True in DAG definition |
Data Sharing & Communication
| Term | Definition | Usage |
|---|---|---|
| XCom | "Cross-communication" — mechanism for tasks to exchange small messages. | ti.xcom_push(key="row_count", value=1000) |
| Connection | Stored credentials for connecting to external systems (DB, API, cloud). | Managed via UI → Admin → Connections |
| Variable | Global key-value config available to all DAGs. | Variable.get("environment") |
| Pool | A mechanism to limit the number of concurrent tasks of a certain type. | pool="database_connections" with 5 slots |
Infrastructure Components
| Term | Definition | Role |
|---|---|---|
| Scheduler | The daemon that monitors DAGs, triggers DAG Runs, and submits tasks. | The "brain" of Airflow |
| Webserver | The Flask app that serves the Airflow UI. | Monitoring & management interface |
| Executor | Determines how tasks are run (locally, on Celery workers, in K8s pods). | CeleryExecutor, KubernetesExecutor |
| Metadata Database | PostgreSQL/MySQL database storing all state, config, and history. | The single source of truth |
| Worker | The process that actually executes tasks. | Runs on Celery workers or K8s pods |
| Triggerer | A daemon that efficiently handles deferred (async) tasks. | New in Airflow 2.2+ |
graph LR
subgraph "Airflow Components"
SCHED["⏰ Scheduler"] --> DB[("💾 Metadata DB")]
WEB["🖥️ Webserver"] --> DB
WORKER["⚙️ Worker"] --> DB
TRIG["⚡ Triggerer"] --> DB
SCHED --> WORKER
SCHED --> TRIG
end
style SCHED fill:#017cee,stroke:#015bb5,color:#fff
style WEB fill:#00c7d4,stroke:#009ea8,color:#fff
style WORKER fill:#00ad46,stroke:#008a38,color:#fff
style DB fill:#FF9800,stroke:#F57C00,color:#fff
style TRIG fill:#9C27B0,stroke:#7B1FA2,color:#fff
💡 Tip
In interviews, you'll almost always be asked: "What are the main components of Airflow?" The answer is: Scheduler, Webserver, Workers, Metadata Database, and (optionally) Triggerer. Know what each one does.
In interviews, you'll almost always be asked: "What are the main components of Airflow?" The answer is: Scheduler, Webserver, Workers, Metadata Database, and (optionally) Triggerer. Know what each one does.
Task States
A Task Instance goes through several states during its lifecycle:
| State | Icon | Meaning |
|---|---|---|
none |
⚪ | Task has not been queued yet |
scheduled |
🟡 | Scheduler has determined it should run |
queued |
🟤 | Task is assigned to an executor and waiting for a worker |
running |
🟢 | Task is currently being executed |
success |
✅ | Task completed successfully |
failed |
🔴 | Task execution encountered an error |
up_for_retry |
🟠 | Task failed and is waiting to be retried |
up_for_reschedule |
🔵 | Sensor is in reschedule mode and waiting |
upstream_failed |
🟣 | An upstream dependency failed |
skipped |
⬜ | Task was skipped (e.g., by BranchOperator) |
deferred |
🔷 | Task is waiting for an external trigger (Triggerer) |